METHOD AND APPARATUS FOR GENERATING STEREOSCOPIC IMAGES 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

5 The present invention relates to a method and 

apparatus for generating stereoscopic images. 

2. Description of the Related Arts 

Among stereoscopic image display devices is that 
which realizes stereoscopic vision by allowing the 

10 observer's right and left eyes to perceive different 
images, thus causing parallax to take place. Such 
stereoscopic vision has heretofore been implemented by the 
lenticular system using lenticular lens (e.g.. Fig. 6.18 
in Document 1) , the parallax barrier system using parallax 

15 barrier (e.g.. Fig. 6.15 of Document 1, Document 2) and 
others . 

Document 1 

'''' Fundament als to 3D Picture" supervised by Takehiro 
Izumi published by Ohmsha, 1995.6.5 (pp. 145-150) 
20 Document 2 

Japanese Patent No. 3096613 

In the aforementioned parallax barrier system^ a 
parallax barrier made of a number of fine slits is attached 
to limit the viewable direction for each pixel of the 
25 stereoscopic display device. 

That is, images for right and left eyes that cause 
binocular parallax are set up in a single flat display such 
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that they are perceived by corresponding eyes. 
Implementation of stereoscopic image display through such 
binocular parallax requires image data for right and left 
eyes to be created. Further, trinocular or more 
5 multinocular stereoscopic image display requires image 
data for a corresponding number of eyes to be created. 

In a device that displays multinocular stereoscopic 
images, therefore, the numbers of times coordinate 
conversion processing is performed and a memory is 

10 accessed increase with the number of viewpoints. To 

resolve such an inconvenience, a method has been suggested 
in which images corresponding to a plurality of viewpoints 
are created by placing a virtual viewpoint in a space and 
displacing screen system objects based on the virtual 

15 viewpoint in screen coordinates according to binocular 
parallax (e.g.. Document 3). 
Document 3 

Japanese Patent Application Laid-open No . 2002-73003 
In the case of stereoscopic display based on binocular 
20 parallax, there exists a predetermined range in which 
stereoscopic vision is possible with reference to the 
image display surface. Outside the stereoscopic viewable 
range, the observer cannot achieve stereoscopic vision, 
perceiving the image as being shaky. This will 
26 substantially burden the observer' s eyes if the image is 
continuously observed . 

This will be described further with reference to Figs . 
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lA through IF. Fig. lA illustrates a view from above of 
a case in which images for left and right eyes are captured 
with parallax cameras CL and CR respectively for left and 
right eyes and having parallaxes when an object 1 serves 
5 as a viewpoint OP for an image consisting of an object 2 
arranged on the front and an object 3 on the back of the 
ob j ect 1 . 

At this time, coordinate data SL for left eye and SR 
for right eye obtained respectively by the parallax 

10 cameras CL for left eye and CR for right eye are as shown 
in Figs. IB and IC. 

Fig. ID illustrates image data SL and SR for left and 
right eyes corresponding respectively to the coordinate 
data SL and SR for left and right eyes. An observer 5 

15 observes the image data SL and SR for left and right eyes 
as the data is displayed on a stereoscopic image display 
surface SC of a display device using the barrier system, 
the lenticular system or other system. 

The observer 5 can perceive the displayed image data 

20 SL and SR for left and right eyes as stereoscopic image 
by sensuously combining the two pieces of data. 

If the objects 2 and 3 form their images at or more 
than a predetermined distance (a range 4 that gives 
stereoscopic perception) from the stereoscopic image 

25 display surface SC of the display device, the images of 
the objects 2 and 3 observed by the left and right eyes 
of the observer 5 undergo considerable displacements of 
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corresponding points, {2-1, 2-2) (3-1, 3-2), thus 
resulting in being perceived as shaky and making 
stereoscopic vision impossible. In the example shown in 
Figs. 1, the image with only the object 1 is 
5 stereoscopically viewable. 

A critical visual factor for achieving stereoscopic 
vision relates to binocular parallax. The fact that right 
and left eyes are apart prevents the same image from being 
perceived by both eyes when a certain object is looked at, 

10 causing a discrepancy at a position more distant than the 
gazing point. In the presence of discrepancy between 
images perceived by two eyes, the images are generally 
viewed as a double image. However, if binocular parallax 
is equal to or smaller than a certain level, the images 

15 are merged, resulting in being perceived as a 3D image. 

Fig. 2 illustrates an explanatory drawing thereof. 
In Fig. 2, we let an observation distance from the observer 
5 to the display surface SC be Lreai/ an eye~to-eye distance 
of the observer 5 be E, a limit distance from the display 

20 surface SC to the forward stereoscopic viewable range 4 
be n, a limit distance from the display surface SC to the 
backward stereoscopic viewable range 4 be f, a difference 
in displacement between corresponding points due to 
parallax be D (a difference in displacement due to parallax 

25 that gives forward stereoscopic viewable image formation 
limit be Dn and a difference in displacement due to parallax 
that gives backward stereoscopic viewable image formation 
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limit be Df) . 

For most observers, a physiological limit distance 
for binocular fusion is roughly 0.03 times the observation 
distance Lreai • For instance, if the observation distance 
5 Lreai = 60cm, it becomes difficult to s ter eoscopical ly view 
the corresponding point at a distance of 1.8cm or more in 
the difference in displacement Dn or Df . 

In this case, if we let the observer's eye-to-eye 
distance E be 6.5cm, the forward image formation limit n 

10 . is located n « 13.0cm from the display surface SC because 
of the relation n(60 - n) = 1.8/6.5. On the other hand, 
the backward image formation limit f is located f « 23.0cm 
from the display surface SC because of the relation f/ (60 
+ f) = 1.8/6.5. Thus, stereoscopic vision is difficult 

15 outside the stereoscopic viewable range 4 relative to the 
eye-to-eye distance E. 

Such a range in which stereoscopic vision is not 
possible is described neither in the above Document 1 nor 
in the Documents 2 and 3. Therefore, there exist no 

20 descriptions suggesting techniques for addressing such a 
range . 

SUMMARY OF THE INVENTION 
In view of the foregoing, it is an object of the 
present invention to provide a method and apparatus for 
25 generating stereoscopic images that can efficiently 
generate stereoscopic images that do not burden the 
observer's eyes. 
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It is another object of the present invention to 
provide a method and apparatus for generating stereoscopic 
images for making the stereoscopic images more highlighted 
on the screen by displaying, from a different viewpoint, 
5 stereoscopic and planar images in a mixture. 

In order to attain the above objects, a method and 
apparatus for generating stereoscopic images according to 
the present invention include, as a first aspect, 
converting, of objects made of polygons having 3D 

10 coordinates, object data to be displayed in a planar view 
to reference camera coordinate system data with its origin 
at a reference camera and converting object data to be 
displayed in a stereoscopic view to parallax camera 
coordinate system data for right and left eyes 

15 respectively with their origins at parallax cameras for 
right and left eyes having predetermined parallax angles; 
drawing the reference camera coordinate system object data 
and the parallax camera coordinate system object data for 
right eye as image data for right eye in a video memory; 

20 drawing the reference camera coordinate system object data 
and the parallax camera coordinate system object data for 
left eye as image data for left eye in the video memory; 
and synthesizing the image data for right and left eyes 
drawn in the video memory and displaying, on a stereoscopic 

25 display device, images mixing stereoscopic and planar 
ob j ect s . 

As a second aspect, to attain the above objects, in 
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the method and apparatus for generating stereoscopic 
images according to the first aspect of the present 
invention, the objects to be displayed in a planar view 
are objects having their image formation positions outside 
5 a stereoscopic viewable range of the stereoscopic display 
device in a 3D coordinate space. 

In order to attain the above objects, a method and 
apparatus for generating stereoscopic images according to 
the present invention comprise, as a third aspect, 

10 converting object data made of polygons having 3D 

coordinates to parallax camera coordinate system data 
respectively with their origins at parallax cameras for 
right and left eyes having predetermined parallax angles; 
performing scaling using the converted parallax camera 

15 coordinate system data to compress coordinates of the 
parallax camera coordinate system data in the direction 
of the depth of a stereoscopic viewable range of a 
stereoscopic display device such that all the objects have 
their image formation positions within the stereoscopic 

20 viewable range; drawing the scaled parallax camera 

coordinate system data in a video memory; and displaying, 
on the stereoscopic display device, drawing data drawn in 
the video memory. 

In order to attain the above objects, a method and 

25 apparatus for generating stereoscopic images according to 
the present invention comprise, as a fourth aspect, 
converting object data made of polygons having 3D 
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coordinates to parallax camera coordinate system data 
respectively with their origins at parallax cameras for 
right and left eyes having parallax angles; narrowing the 
parallax angles during conversion to the parallax camera 
5 coordinate system data such that all objects of the 

parallax camera coordinate system data to be converted 
have their image formation positions within a stereoscopic 
viewable range of a stereoscopic display device; and 
displaying, on the stereoscopic display device, the 

10 converted parallax camera coordinate system data at the 
narrowed parallax angles. 

In order to attain the above objects, a method and 
apparatus for generating stereoscopic images according to 
the present invention comprises, as a fifth aspect, 

15 converting object data made of polygons having 3D 

coordinates to reference camera coordinate system data 
with its origin at a reference camera; converting, of 
object data converted to the reference camera coordinate 
system data, object data to be displayed in a stereoscopic 

20 view to parallax camera coordinate system object data 
respectively with their origins at parallax cameras for 
right and left eyes having predetermined parallax angles; 
drawing the reference camera coordinate system object data 
and the parallax camera coordinate system object data for 

25 right eye as image data for right eye in a video memory; 
drawing the reference camera coordinate system object data 
and the parallax camera coordinate system object data for 
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left eye as image data for left eye in the video memory; 
and 

synthesizing the image data for right and left eyes 
drawn in the video memory and displaying, on a stereoscopic 
5 display device, images mixing stereoscopic and planar 
ob j ect s . 

As a sixth aspect, to attain the above objects, in 
the method and apparatus for generating stereoscopic 
images according to the fifth aspect of the present 

10 invention, the objects to be displayed in a planar view 
are objects having their image formation positions outside 
a stereoscopic viewable range of the stereoscopic display 
device in a 3D coordinate space. 

In order to attain the above objects, a method and 

15 apparatus for generating stereoscopic images according to 
the present invention comprises, as a seventh aspect, 
converting object data made of polygons having 3D 
coordinates to reference camera coordinate system data 
with its origin at a reference camera; generating, from 

20 the reference camera coordinate system data, parallax 
camera coordinate system data respectively with their 
origins at parallax cameras for right and left eyes having 
parallax angles; performing compression scaling during 
generation of the parallax camera coordinate system data 

25 such that all objects have their image formation positions 
within a stereoscopic viewable range of a stereoscopic 
display device; drawing the parallax camera coordinate 
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system data for right and left eyes in a video memory; and 
synthesizing the image data for right and left eyes drawn 
in the video memory and displaying the data on the 
stereoscopic display device. 
5 In order to attain the above objects, a method and 

apparatus for generating stereoscopic images according to 
the present invention comprises, as an eighth aspect, 
converting object data made of polygons having 3D 
coordinates to reference camera coordinate system data 

10 with its origin at a reference camera; converting the 
reference camera coordinate system data to parallax camera 
coordinate system data respectively with their origins at 
parallax cameras for right and left eyes having parallax 
angles; narrowing the parallax angles during conversion 

15 to the parallax camera coordinate system data such that 
all objects of the parallax camera coordinate system data 
to be converted have their image formation positions 
within a stereoscopic viewable range of a stereoscopic 
display device; and displaying, on the stereoscopic 

20 display device, the converted parallax camera coordinate 
system data at the narrowed parallax angles. 

As a ninth aspect, to attain the above objects, in 
the method and apparatus for generating stereoscopic 
images according to any one of the first to eighth aspects 

25 of the present invention, the parallax angles of the 

parallax cameras are adjustable in real time by operations 
of an observer. 
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As a tenth aspect, to attain the above objects, in 
the method and apparatus for generating stereoscopic 
images according to the ninth aspect of the present 
invention, the parallax angles are continuously and 
5 gradually varied as a result of the adjustment by 
operations of the observer. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The above and other objects, aspects, features and 
advantages of the present invention will become more 
10 apparent from the following detailed description when 
taken in conjunction with the accompanying drawings, in 
whi ch^ 

Figs. lA through IF illustrate a conventional 
example ; 

15 Fig. 2 illustrates a stereoscopic viewable range 4 

shown in Figs. 1; 

Figs. 3A through 3F illustrate a first solution 
principle of the present invention; 

Figs. 4A through 4C illustrate another solution 
20 principle of the present invention; 

Figs. 5A through 5F illustrate a method according to 
a third solution principle of the present invention; 

Figs. 6A and 6B illustrate a general view of a 
configuration example for a gaming apparatus as an 
25 apparatus for generating stereoscopic images to which a 
method for generating stereoscopic images according to a 
solution principle of the present invention is applied; 
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Fig, 7 illustrates a block diagram showing a 
configuration of the apparatus for generating 
stereoscopic images to which the method for generating 
stereoscopic images according to the solution principle 
5 of the present invention is applied; 

Fig. 8 illustrates a flowchart showing processing of 
the geometry unit 14 that provides the features of the 
method for generating stereoscopic images of the present 
invention; 

10 Figs. 9A through 9D illustrate processing steps 

corresponding to Fig. 8; 

Figs. lOA through IOC illustrate a method for 
converting reference camera coordinate system data to 
parallax camera coordinate system data to generate 
15 parallax images; 

Fig. 11 illustrates a configuration example for a 
parallax conversion unit; 

Fig. 12 illustrates a working example for configuring 
the parallax conversion unit with an operator; 
20 Fig. 13 illustrates a working example for speeding 

up processing of the parallax conversion unit; 

Figs. 14A through 14C illustrate explanatory 
drawings describing a difference in displacement D due to 
parallax ; 

25 Figs. 1 5A and 15B illustrate explanatory drawings 

describing changing of applied parallax data by a parallax 
adjustment unit 103; 



Fig. 16 illustrates an example of processing 
operations in Fig. 7 corresponding to Figs. 15; 

Fig. 17 illustrates a working example in which only 
objects in the air are viewed s tereoscopical ly while an 
object on the ground is viewed planarly; 

Fig. 18 illustrates a plan view corresponding to Fig. 

17; 

Fig. 19 illustrates a stereoscopic/planar image 
mixture drawing routine flow; 

Fig. 20 illustrates a drawing routine flow for right 
(left) eye; 

Figs. 21A through 21C illustrate explanatory 
drawings describing a synthesized image for stereoscopic 
viewing in the working example shown in Fig. 17; and 

Figs. 22A through 22E illustrate the process of 
displaying drawn images for left and right eyes, described 
in Figs. 17 to 21, on a stereoscopic display device. 
DESCRIPTION OF THE PREFERRED EMBODIMENTS 

While embodiments of the present invention will be 
described below with reference to the accompanying 
drawings, the solution principles of the present invention 
will be described first. 

Figs. 3A through 3C illustrate explanatory drawings 
of a first solution principle of the present invention. 
Fig. 3A illustrates a top view showing the objects 2 and 
3 each made of a plurality of polygons that are arranged 
respectively on the front and back of the object 1, that 
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is similarly made of a plurality of polygons, in a 3D 
virtual space. 

The figure illustrates a top view showing a case in 
which, when the object 1 is the viewpoint OP, images for 
5 left and right eyes are captured with the parallax cameras 
CL and CR respectively for left and right eyes, each of 
which has a line of sight at a predetermined angle relative 
to a line of sight from a reference camera RC toward the 
viewpoint OP. 

10 We now consider a case in which the objects 2 and 3 

are displayed in a planar view while the object 1 is 
displayed in a stereoscopic view. In this case, 
coordinate data of the objects 2 and 3 is obtained from 
the reference camera RC . 

15 On the other hand, coordinate data of the object 1 

for left eye is obtained from the parallax camera CL for 
left eye. Similarly, coordinate data of the object 1 for 
right eye is obtained from the parallax camera CR for right 
eye . 

20 The coordinate data of the objects 2 and 3 obtained 

from the reference camera RC is shared as coordinate data 
for left and right eyes. When the objects 1, 2 and 3 are 
positioned as shown in Fig. 3A, therefore, coordinate data 
for left eye is as shown in Fig. 3B while that for right 

25 eye as shown in Fig. 3C. 

The image data SL and SR for left and right eyes, 
obtained respectively from the coordinate data for left 



and right eyes, is as shown in Fig. 3D. 

The image data SL and SR for left and right eyes is 
displayed on a common stereoscopic image display device. 
Fig. 3E illustrates a relation diagram viewed from above 
5 at this time while Fig. 3E a relation diagram viewed from 
the observer 5. 

In Figs. 3E and 3F, the objects 2 and 3 are displayed 
as planar images on the display surface SC of the 
stereoscopic display device while the object 1 is 

10 displayed as a stereoscopic image. This results in the 
image of the object 1 appearing more highlighted than the 
images of the objects 2 and 3. At the same time, as is 
apparent from Fig. 3F, it is possible to prevent the 
displayed images 2 and 3 from appearing shaky as compared 

15 with Fig. IF by displaying the objects 2 and 3 as planar 
images, even if the coordinate positions of the objects 
2 and 3 are outside the stereoscopic viewable range 4. 

If the solution principle is applied, for example, 
to game program images, the peripheral objects 2 and 3 are 

20 displayed non- three-dimensionally as opposed to the 

central object 1. However, since the main object 1 at the 
center can be st ereoscopical ly viewed, game players can 
observe the powerful object 1 image on the whole while 
playing the game. 

25 Figs. 4A through AC illustrate a second solution 

principle of the present invention. Fig. 4A illustrates 
a top view showing a case in which, when the object 1 is 
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the viewpoint OP, the object 1 placed in a virtual space, 
with the objects 2 and 3 arranged respectively on the front 
and back of the object 1, is captured with the parallax 
cameras CL and CR respectively for left and right eyes. 
5 At this time, the objects 2 and 3 are outside the range 

4 that gives three-dimensional appearance on the display 
device. In such a case, the second solution principle 
scales all objects to compress the coordinate in the 
direction of the depth of the stereoscopic viewable range 

10 4, that is, the coordinate along the Z axis of the virtual 
space such that the images of the objects 2 and 3 are inside 
the stereoscopic viewable range 4 that gives 
three-dimensional appearance on the display device (refer 
to Fig. 4B) . This allows for the objects 1, 2 and 3 to 

15 be observed without changing the relative positional 

relationship between the objects, as shown in Fig. 4C. 

However, when the objects in the virtual space are 
scaled, it is necessary to recalculate vertex positions 
of the polygons constituting the objects, thus resulting 

20 in increased amount of processing. In this respect, a 
third solution principle shown in Figs. 5A through 5F is 
preferred . 

Fig. 5A illustrates a top view showing a case in which, 
when the object 1 is the viewpoint OP, an image of the object 
25 1, with the objects 2 and 3 arranged respectively on the 
front and back of the object 1, is captured with the 
parallax cameras CL and CR for left and right eyes having 
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parallax angles. 

The image data SL and SR for left and right eyes, 
obtained at this time respectively from the parallax 
cameras CL and CR for left and right eyes for the projection 
5 surface SC, is as shown in Figs. 5B and 5C . Further, Fig. 
5D illustrates images for left and right eyes generated 
from the image data SL and SR for left and right eyes. 

The feature of the solution principle shown in Fig. 
5E is that the parallax angle between the parallax cameras 
10 CL and CR for left and right eyes is small enough such that 
the objects 2 and 3 fall within the stereoscopic viewable 
range 4 . 

This reduces the margin of displacement as a result 
of parallax, thus reducing the distance from the image 

15 display surface SC to the image formation positions of the 
objects 2 and 3 and thereby allowing for the objects 2 and 
3 to be placed inside the stereoscopic viewable range 4. 
Therefore, the solution principle provides the same effect 
as that discussed above in which the objects are scaled. 

20 That is, the objects 1, 2 and 3 can be stereoscopically 

viewed without changing the relative positional 
relationship between the objects in the scene as a whole. 

Figs. 6A and 6B illustrate a configuration example 
for a gaming apparatus ICQ as an apparatus for generating 

25 stereoscopic images to which the method for generating 
stereoscopic images according to the aforementioned 
solution principle of the present invention is applied. 
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Fig. 6A illustrates a general view of the configuration 
example for the gaming apparatus 100 while Fig. 6B a 
hardware block diagram. 

The gaming apparatus 100 is provided with an operating 
5 console projecting to the front of an enclosure 101, and 
the operating console is provided with a game control unit 
102, a parallax adjustment unit 103 and further a 
stereoscopic image display unit 104 that faces forward. 
Further, the gaming apparatus 100 incorporates an 
10 arithmetic and image processing unit 105. 

The arithmetic and image processing unit 105 
generates stereoscopic image data and displays the data 
on the stereoscopic image display unit 104 according to 
information input from the game control unit 102 and the 
15 parallax adjustment unit 103. 

Fig. 7 illustrates a block diagram showing a 
configuration example for the arithmetic and image 
processing unit 105 that is provided inside the enclosure 
101 of the gaming device 100 and the method for generating 
20 stereoscopic images according to the solution principle 
of the present invention is applied. 

In Fig. 7, a work memory 10 stores an application 
program while a display list memory 11 stores a display 
list - a program that handles setup, arithmetic and polygon 
25 drawing procedure to create models. 

The application program and the display list are read 
from the work memory 10 for program processing in a CPU 
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12. The program processing results by the CPU 12 are sent 
to a geometry unit 14 via a bridge 13 - an interface. 

Based on program processing results by the CPU 12, 
the geometry unit 14 converts model data made of a 
5 plurality of polygons defined by world coordinate data to 
camera coordinate system data with its origin at a camera 
position and further performs processing such as clipping, 
culling, brightness calculation, texture coordinate 
arithmetic and perspective projection transform. In 

10 converting model data defined by world coordinate data to 
camera coordinates, in particular, parallax conversion - 
a feature of the present invention- is performed after 
conversion to reference camera coordinate system data, as 
a result of which parallax camera coordinate system data 

15 for right and left eyes is obtained. 

Next, a renderer (rendering unit) 15 reads texture 
data from a video RAM 16 that serves both as a texture memory 
and a frame buffer and fills the polygons based on the 
texture coordinate arithmetic results. 

20 Image data with filled texture data is stored again 

in the video RAM 16, with reference camera coordinate 
system data and parallax camera coordinate system data for 
right eye used as image data for right eye and reference 
camera coordinate system data and parallax camera 

25 coordinate system data for left eye used as image data for 
left eye. Then, a display controller 17 synthesizes image 
data for right and left eyes read from the video RAM 16, 
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and the synthesized image data is sent to a stereoscopic 
display device 18 for display of a stereoscopic image. 

Fig. 8 illustrates a flowchart showing processing of 
the geometry unit 14 that provides the features of the 
5 method for generating stereoscopic images of the present 
invention. Figs. 9 illustrate processing steps 
corresponding to Fig. 8. 

Note that processing may be performed on a 
polygon-by-polygon basis or ver t ex-by- vert ex basis in Fig. 
10 8 . 

First, model data 20 having models 1 and 2 and stored 
in work memory 11 is, for example, read into the geometry 
unit 14 via the bridge 13 under the control of the CPU 14 
in Fig. 7 (processing step PI) . 

15 The model data has local coordinates. Therefore, the 

local coordinate system model data is converted by the 
geometry unit 14 to the world coordinate system model data 
20 as shown in Fig. 9A and is further subjected to 
coordinate conversion from world coordinate system data 

20 to reference camera coordinate system data with its origin 
at the reference camera RC (processing step P2) . 

Model data 14-1 converted to reference camera 
coordinate system data through coordinate conversion is 
then subjected to parallax conversion (processing step P3) , 

25 transforming the data into parallax camera coordinate 
system data 14-2. Fig. 9B illustrates the models 1 and 
2 in the reference camera coordinate system with its origin 
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at the reference camera RC while Fig. 9C the models 1 and 
2 in the parallax camera coordinate system with its origin 
at a parallax camera R' C that is at a parallax angle 0 
relative to the line of sight of the reference camera RC . 

While only one parallax camera, the parallax camera 
R'C, is shown in Fig. 9C for simplicity of description, 
at least two parallax cameras are required that form the 
predetermined parallax angle 0 in the directions of left 
and right eyes relative to the reference camera RC . 

Fig. 9D illustrates a relation between the reference 
camera coordinate system and the parallax camera 
coordinate system . 

Next, the parallax camera coordinate system data 14-2 
is subjected to perspective projection transform 
(processing step P4), as a result of which projection 
coordinate system data 14-3 or a 2D screen coordinate 
system is obtained. 

Then, the projection coordinate system data 14-3 is 
output to the rendering unit 15 that draws parallax image 
data in the video memory 16. 

In the above description, the feature of the present 
invention differs from that of the method for generating 
image data described in cited Document 1 in that the 
parallax camera coordinate system data 14-2 is obtained 
by conversion from the reference camera coordinate system 
data 14-1 before the reference camera coordinate system 
data 14-1 is subjected to perspective projection transform 
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(processing step P3). 

Further, during conversion to the parallax camera 
coordinate system data (processing step P3), processing 
is performed in correspondence with the principles of the 
5 present invention shown in Figs. 3 to 5; switching between 
the parallax camera coordinate system data and the 
reference camera coordinate system data such that the 
image formation positions of the objects fall within the 
stereoscopic viewable range of the stereoscopic display 

10 device 18 (refer to Figs. 3A through 3F) , scaling of the 
parallax camera coordinate system data (refer to Figs. 4A 
through 4C) and setting of a small parallax angle (refer 
to Figs. 5A through 5F) . 

A method will now be described below with reference 

15 to Figs. lOA through IOC for converting the reference 
camera coordinate system data 14-1 to the parallax camera 
coordinate system data 14-2. 

As shown in Fig. lOA, if coordinate origins are at 
the reference camera RC, an object having coordinates P 

20 (x, y, z) is seen as located at coordinates P' (x' , y' , 
z' ) when we let the distance to the viewpoint OP (point 
where the line of sight from the parallax camera R'C 
intersects with that from the reference camera RC) be 
Lvirtuai and the parallax angle relative to the reference 

25 camera RC be 0. 

At this time, the following relationship holds: 
Equation 1 
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x' = xcosG + z (~sin0) + Lvirtuaisin9 

y' = y Equation 1 

z' = xsinG + 2cos9 + Lvirtuai (1 - cos9) 
Here, the parallax camera R' C position can be 
approximated as shown below if the parallax camera R'C is 
assumed to be on the X axis that includes a position 
coordinate of the reference camera RC as shown in Fig. 9D 
and if the variation along the Z axis due to parallax is 
ignored . 
Equation 2 

x' = XCOS0 - zsinG + LvirtuaisinB 

y' ±^ y Equation 2 

z ' ^ z 

From the equation 2, the coordinates P (x, y, z) as 
seen from the reference camera RC can be approximately 
converted to the coordinates P' (x' , y' , z') as seen from 

the parallax camera R' C using a parameter (Lvirtuai/ G)- 
By subjecting polygon vertices of all model data to 
this conversion, a scene as seen from the reference camera 
RC can be approximately converted to a scene as seen from 
the parallax camera SC (the conversion and the parameter 
used are hereafter referred respectively to as parallax 
conversion and parallax parameter) . 

By setting a parameter (1) (Lvirtuai/ - 9) as the 
parallax parameter for left eye and a parameter (2) (L virtual / 
9) as the parallax parameter for right eye, binocular 
parallax images can be generated for a binocular 
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stereoscopic display device as shown in Fig. lOB. In the 
case of quadranocular images, the parameter set consists 

of (1) (Lvirtual/ ~ 39) , (2) (Lvirtual/ - 0) , (3) (Lvi rt ua 1 / 

and (4) (Lvirtuai/ 30) as shown in Fig. IOC. Similarly, 
5 expansion to multinocular images for an arbitrary number 
n of eyes is readily possible. 

The parallax conversion is carried out by providing 
a parallax conversion unit 140 in the geometry unit 14 as 
shown in Fig. 11. That is, parallax conversion arithmetic 
10 142 can be performed with parallax conversion parameter 

(Lvirtual/ n9) 141 according to the equations 1 and 2 by 
inputting reference camera coordinate system data and by 
providing hardware or software. 

As described above, the parallax camera coordinate 
15 system data P' (x'' , y' , z'), obtained by subjecting the 
reference camera coordinate system data P (x, y, z) to 
parallax conversion with the parallax conversion 

parameter P (Lvirtuai/ ©) / is expressed, from the equation 
2, as shown below. 
20 x' = xcosG - zsinG + LvirtuaisinG 

y' = y 

z ' =^ z 

Therefore, performing the conversion on only the x 
component and substituting A = cosG, B = -sinG and C = 
25 LvirtuaisinG into the parallax conversion parameter P 
(Lvirtual/ G) for further reduction in arithmetic cost 
yields : 



x' =Ax+Bz+C 

By exploiting the above-described advantage, the 
parallax conversion unit 140 shown in Fig. 11 can be 
configured with an operator having a simple configuration 
5 as shown in Fig. 12. 

Further review reveals that storing parallax 
parameters 141-1 to 141-n for n number of eyes in the 
parallax conversion unit 140 as shown in Fig. 13 allows 
for conversion of a single piece of reference camera 
10 coordinate system data to parallax camera coordinate 

system data for n the number of eyes, thus speeding up 
processing since model data readout (processing step PI 
in Fig. 8) and coordinate conversion in the geometry unit 
14 (processing step P2 in Fig. 8) can be performed in 
15 parallel and in one operation. 

A method will be described next for determining a 
parallax parameter used for the solution principle shown 
in Fig. 4. 

A general equation of perspective projection 
20 transform (x, y, z) (Sx, Sy) for converting 3D 

coordinates to 2D screen coordinates is expressed as 
follows : 

Sx = FXx/z + Ch 
Sy = Fxy/z + Cv 
25 (where F: focus value, Ch: horizontal center value, Cv: 
vertical center value) 

If we let corresponding points, converted using the 
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parallax conversion parameters (Lvirtuai/ ©) and (Lvirtuai/ 
-6) and provided with parallax by the parallax cameras CR 
and CL for right and left eyes, be (Xr, y, z) and (Xl/ Y, 
z) , the difference in displacement D on the display screen 
5 of a stereoscopic display device 19 is as follows: 
Equation 3 

D = ISxR - SxlI 

= I Fxr/z + Ch - (Fxl/z + Ch) | 

= |F(xcos9 - zsinG + Lvirtuais inG ) / z - F{xcos(-G) - 
10 2sin(-G) + Lvirtuaisin (-G) } /z I 

= |F(xcosG - zsinG + LvirtuaiS inG ) / z - F(xcosG + zsinG 

— LivirtualSl nG) /z I 

= I 2FsinG (Lvirt.nal - z)/z| 

= I 2FsinG (Lvirtuai/z - 1)| 
15 For the range of z>0, 

(i) 0 < Z < Lvirtuai : D = 2FsinG (Lvirtuai - z)/z 

(ii) z = Lvirtuai : D = 0 

(iii) Lvirtuai < Z : D = 2FsinG(Z - Lvirtuai) /z 

Next, if the distance Lvirtuai from the observer 5 to 
20 the image display screen SC and the eye-to-eye distance 
E of the observer 5 in a real space are fixed as shown in 
Figs. 14, the distance from the image display screen SC 
to the object image formation position is determined by 
the difference in displacement D due to object parallax. 
25 That is, it is only necessary to set the difference in 
displacement D due to parallax such that the image 
formation position falls within the stereoscopic viewable 



)^Equation 3 
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range 4 . 

If we let the distance from the observer 5 to the 
display surface SC be Lreai/ the eye-to-eye distance of the 
observer 5 be the distance from the display surface SC 
5 to the forward stereoscopic viewable range 4 be n, the 
distance from the display surface SC to the backward 
stereoscopic viewable range 4 be f, the difference in 
displacement between corresponding points due to parallax 
be D, the difference in displacement due to parallax that 

10 gives forward stereoscopic viewable image formation limit 
be Dn and the difference in displacement due to parallax 
that gives backward stereoscopic viewable image formation 
limit be Df, the forward merging limit that occurs when 
D = Dn is as follows from the triangle similarity 

15 relationship: 

Dn/n = E/ (Lreal - H) 
Dn = Exn/ (Lreal " n) 

From equation 3 (i) , the following relationship holds 

between 9 and z: 
20 2Fsin9 (Lvirtuai - z)/z = Exn/(Lreai - n) 

sinG (Lvirtuai - z)/z = Exn/ [ 2F (L^eai - n) ] 

If we let the forward limit of the target display 

region in a 3D coordinate space be the forward clipping 

surface or z = Cn/ then 9 = Gnear that satisfies the following 
25 is an angle necessary for merging the forwardmost 
displayed object: 

Sin9 (Lvirtuai - Cn)/Cn = Exn / [ 2 F ( Lrea 1 " H)] 



sinG = Exnxcn/[2F(L real ~ H. ) (Lvirtual ~ Cn ) ] 

On the other hand, the backward merging limit that 
occurs when D = Df is as follows from the triangle 
similarity relationship : 

5 Df/f = E/ (Lreal + f ) 

Dn = Exf / (Lreal + f ) 

From equation 3(iii), the following relationship 
holds between 0 and z: 

2Fsin0(2 - Lvirtual) /Z = Exf / (Lreal + f) 
10 Sine(Z - Lvirtual) /Z == Exf / [2F (Lreal + f)] 

If we let the backward limit of the target display 
region in a 3D coordinate space be the backward clipping 
surface or z = Cf, then 0 = Gfar that satisfies the following 
is an angle necessary for merging the backwardmost 
15 displayed object: 

Sin0(Cf — Lvirtual ) /Cf = Exf/ [2F (L 

real ' / J 

sin0 = Exfxcf/[2F(L 

real + f) (Cf - -L»vi rt ua 1 7 J 

Hence, a parameter 0 that allows merging of all 
objects for Cn ^ z ^ Cf is 

20 0 = min[0near, 0far] 

When 0near = 0far/ the following relationship holds: 

F (Lreal " n) / [n (Lreal + f ) ] = Cn (Cf - Lvirtual) / [ Cf (Lvirtual 

- Cn) 

Also, when Dn = Df, the following relationship holds: 

25 (Lreal " n)/n = (Lreal + f)/f 

L/2x (1/n - 1/f ) =1 
Therefore, when 0near = 0far and Dn = Df 
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Cn(Cf - Lvirtual)/ Cf(Lvirtual ~ Cn ) ] = 1 
Lvirtual = 2CnCf/ ( Cn + Cf) 

At this time, 

SinGnear = sinGfar = Exfx(Cn + Of ) / [ 2 F { Lreal + f) (Cf - 

5 Cn)] 

Incidentally, if we let Lvirtuai = Lreai/ Cn = L - n and 

Cf = L + f, then sinG near ~ SinGfar ~ E/ (2F) reSUltS. 

The parallax parameter G can be found as described 
above. Note that Lvirtuai can be found from the gazing point 

10 (point of intersection of lines of sight of the parallax 
cameras) and the distance to the reference camera. 
Although, in the above description, use of hardware was 
mainly discussed for acquisition of parallax camera 
coordinate data from reference camera coordinate data, 

15 software may be used, if attention is focused on the 
feature of the present invention for displaying 
stereoscopic and planar images in a mixture, to directly 
obtain parallax camera coordinate data for left and right 
eyes without being based on reference camera coordinate 

20 data. 

Physiological factors for stereoscopic perception 
are different between the observers 5. Further, the 
degree of stereoscopic perception varies depending on the 
image displayed during game playing. Therefore, the 
25 gaming apparatus shown in Figs. 6 is provided with the 
parallax adjustment unit 103 in correspondence therewith. 

That is, the player can change parallax angle data 

29 



properly in real time by operating the parallax adjustment 
unit 103 during parallax conversion (processing step P3) 
even when the game is in progress* 

In this case, it is possible for the observer to 
5 perceive three-dimensionality suited for him or her. In 
particular, if the gaming apparatus is installed in an 
environment such as a game center where an indefinite 
number of people can become players, it is preferred that 
the parallax adjustment unit 103 be provided such that the 

10 parallax angle can be adjusted suitably for physiological 
factors of each player, instead of automatically using the 
same parallax angle. It is further preferred that the 
parallax angle be changed gradually from weaker to 
stronger three-dimensionality or continuously. 

15 Figs. 15 A and 15B illustrate explanatory drawings 

describing changing of applied parallax data by the 
parallax adjustment unit 103 while Fig. 16 illustrates an 
example of processing operations corresponding to Figs. 
15. Fig. 15A illustrates a case in which the space between 

20 the reference camera RC and the parallax camera R'C is 
narrow while Fig. 15B a case in which the space between 
the reference camera RC and the parallax camera R'C is 
wide . 

When the CPU 12 detects a parallax change input from 
25 the parallax adjustment unit 103 (Fig. 16: Yes answered 
in processing step P3-1), the CPU 12 changes applied 
parallax data such as distance between parallax cameras 
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(processing step P3-2) . The CPU 12 continuously and 
gradually brings the parallax camera position closer to 
the camera position corresponding to the applied parallax 
data until the current parallax camera position matches 
5 that based on the applied parallax data (processing steps 
P3-3, P3~4) . 

It is important to gradually bring the parallax camera 
position closer to the camera position corresponding to 
the applied parallax data for maintaining binocular fusion 

10 (state in which the observer is capable of stereoscopic 
vision) particularly if the space between parallax cameras 
is increased. That is, since instantaneous transition 
from weak to strong parallax states is likely to throw 
binocular fusion off balance, gradually expanding the 

15 space between parallax cameras prevents such an 
inconvenience . 

In Figs. 15, the parallax camera R' C position is 
adjusted from Fig. 1 5A to Fig. 15B or vice versa. With 
the position shown in Fig. ISA, the objects 2 and 3 are 

20 close to the stereoscopic display surface SC (Fig. 15A, 
b) , making stereoscopic vision easier but resulting in an 
image poor in three-dimensionality. With the position 
shown in Fig. 15B, on the other hand, the objects 2 and 
3 are far from the stereoscopic display surface SC (Fig. 

25 15A, b), making stereoscopic vision more difficult but 
providing an image rich in three-dimensionality. 

Thus, by using parallax adjustment unit 103, it is 
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possible to gradually switch from a state in which 
stereoscopic vision is easy to achieve by the observer to 
an observation state rich in three-dimensionality while 
at the same time maintaining binocular fusion. 
5 Next, Fig. 17 illustrates, as a working example, a 

scene viewed from the camera RC in the sky in which, of 
objects, only objects in the air 110 are viewed 
ster eoscopical ly while an object on the ground 111 is 
viewed planarly. 

10 The example in Fig. 17 shows a state in which only 

the objects in the air 110 are located within the 
stereoscopic viewable range 4, with the object on the 
ground 111 located outside the stereoscopic viewable range 
4, as shown in the corresponding plan view shown in Fig. 

15 18 . 

Figs. 19 and 20 illustrate flowcharts showing 
processing procedures corresponding to the example shown 
in Fig. 17. The objects in the air 110 and the object on 
the ground 111 are assumed to be distinguishable from each 

20 other by the programmer in advance. As for the objects 
in the air 110, the parallax parameters of the parallax 
cameras for right and left eyes are respectively set to 
(Lvirtuai^ 6) and (Lvirtuai/ ) relative to the direction of 
line of sight of the reference camera. As for the object 

25 on the ground 111, the parallax parameters of the parallax 
cameras for right and left eyes are both set to (L virtual / 
0) , that is, brought into agreement with that of the 
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reference camera before a drawing command is issued. 

In response to the drawing command, an image drawing 
routine for right eye Rl and an image drawing routine for 
left eye R2 are executed according to a 
5 stereoscopic/planar image mixture drawing routine flow 
shown in Fig. 19. The drawing routines Rl and R2 are 
executed according to a flow shown in Fig. 20, and the 
sequence of their execution can be changed. 

In the drawing routine flow for right (left) eye shown 
10 in Fig. 20, the position/direction parameters - parallax 

parameters (Lvirtuai/ ©) and (Lvirtuai^ -9) - are set for the 
objects in the air 110 (processing step P20-1), and the 
objects in the air 110 are drawn in the video memory 16 
by the processing performed by the geometry unit 14 and 
15 the rendering unit 15 described in Fig. 8 (processing step 
. P20-2) . 

Further, in the drawing routine flow shown in Fig. 
20, the position/direction parameter (Lvirtuai/ 0) is set 
as the parameter for left (right) eye for the object on 
20 the ground 111 in the same scene (processing step P20-3) , 
and the ob j ect on the ground 1 1 1 i s drawn in the video memory 
16 by the processing performed by the geometry unit 14 and 
the rendering unit 15 described in Fig. 8 (processing step 
P20-3) . 

25 Note that it is possible to reverse the sequence of 

the steps - parameter settings for the objects in the air 
110 and the object on the ground 111 and drawing of the 
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ob j ect s . 

Figs. 21A and 21B illustrate drawn images for right 
and left eyes drawn in the video memory 16 by the above 
drawing routine flows Rl and R2 . 
5 Next, the drawn images of the objects in the air 110 

and the object on the ground 111 for right eye (Fig. 21A) 
and those for left eye (Fig. 21B) drawn in the video memory 
16 by the drawing routines Rl and R2 shown in Fig. 19 are 
synthesized and output to and displayed on the 

10 stereoscopic display device 18. This allows for the 

objects in the air 110 to be displayed in a stereoscopic 
view and the object on the ground 111 to be displayed in 
a planar view. 

Note that since the image of the object on the ground 

15 111 with no parallax is formed on the image display surface 
in Fig. 21C/ the objects in the air 110 are required to 
be located to the front of the camera' s viewpoint in order 
for the objects in the air 110 to be displayed to the front. 
Conversely, placing the objects in the air 110 to the back 

20 of the viewpoint produces an effect similar to deceiving 
picture - the effect that an object that should be on the 
front looks as through it is on the back. 

Figs. 22 illustrate the process of displaying the 
drawn images for left and right eyes, described in Figs. 

25 17 to 21, on the stereoscopic display device 18. 

Figs . 22A and 22B illustrate the drawn images for left 
and right eyes drawn in the video memory based on the 
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drawing data for the objects in the air 110 to be viewed 
stereoscopically and the object on the ground 111 to be 
viewed planarly that are shown respectively in Figs. 21A 
and 21B as examples* That is, one of the images is the 
5 drawn image for left eye (Fig. 22A) resulting from drawing, 
in the video memory 16, the drawing data of the object on 
the ground 111 obtained from the reference camera RC and 
drawing the drawing data of the objects in the air 110 
obtained from the parallax camera for left eye having a 

10 parallax angle relative to the reference camera RC while 
the other image is the drawn image for right eye (Fig. 22B) 
similarly resulting from drawing, in the video memory 16, 
the drawing data of the object on the ground 111 obtained 
from the reference camera RC and drawing the drawing data 

15 of the objects in the air 110 obtained from the parallax 
camera for right eye having a parallax angle relative to 
the reference camera RC . 

These drawn images for left and right eyes are 
tailored to suit the stereoscopic display device to be used. 

20 Figs. 22C and 22D illustrate examples in which the barrier 
system is used for the drawn image for left eye (Fig. 22A) 
and the drawn image for right eye (Fig. 22B) . In these 
examples, a barrier in slit form is formed for each image. 
In the case of Fig. 22C, the image is tailored such that 

25 the slit barrier range cannot be observed with right eye 
while, in the case of Fig. 22D, the image is tailored such 
that the slit barrier range cannot be observed with left 
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eye . 

Next, the images shown in Figs. 22C and 22D are 
synthesized by placing the images one upon another, thus 
generating a synthesized image for stereoscopic viewing 
as shown in Fig. 22E. By displaying the image on the 
stereoscopic display device and observing the image with 
both eyes, it is possible to simultaneously display the 
objects in the air 110 in a stereoscopic view and the object 
on the ground 111 in a planar view on a single screen. The 
synthesis conducted here means tailoring of the images 
such that the image for right eye can be observed only by 
right eye and that the image for left eye only by left eye. 
This technique is applicable to the head mount display 
system in which images for left and right eyes can be 
independently displayed respectively for corresponding 
eyes, to the system in which images for left and right eyes 
are alternately displayed using shutter type glasses and 
further to multinocular stereoscopic display devices. 

As described above with reference to the drawings, 
it is possible, according to the present invention, to 
provide the method and apparatus for generating 
stereoscopic images that can efficiently generate 
stereoscopic images that do not burden the observer' s 
eyes . 

While illustrative and presently preferred 
embodiments of the present invention have been described 
in detail herein, it is to be understood that the inventive 
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concepts may be otherwise variously embodied and employed 
and that the appended claims are intended to be construed 
to include such variations except insofar as limited by 
the prior art. 
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